Questionnaire Modules from STEPS, CINDI, 
CARMEN, BRFSS, TFI, and IPAQ 


STEPS: 

CINDI: 

CARMEN: 

BRFSS 

TFI 

IPAQ 


Stepwise Approach to Surveillance of Noncommunicable 
Disease Risk Factors, WHO 

Countrywide Integrated Noncommunicable Disease 
Intervention Programme, CINDI Health Monitor, WHO 
Regional Office for Europe 

Conjunto de Acciones para la Reduccion Multifactorial de 
Enfermedades No transmisibles, WHO Regional Office for the 
Americas 

Behavioral Risk Factor Surveillance System, Centers for 
Disease Control and Prevention (CDC) 

WHO Tobacco Free Initiative standardised questions 

International Physical Activity Questionnaire, CDC and the 
Karolinska Institute 
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QUESTIONNAIRE DEVELOPMENT 


GOALS AND STANDARDS 

Goals 

• Work toward validated questionnaire sections and items. 

• Minimize respondent burden. 

Standards 

• Ensure that good trend data are obtained by collecting 
consistent information over time. 

• Ensure that data collected are directly linked to prevention use 
(programme planning and evaluation, policy development, etc.). 
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QUESTIONNAIRE DEVELOPMENT 

IMPORTANT CONSIDERATIONS 

Developing Simplicity is key. Questions for obtaining global comparison data 
questionnaire across countries should be few in number, due to cost, respondent 
comparability burden, and difficulty in obtaining general consensus on question 
across countries wording. It is crucial that adequate space is left on the 

questionnaire for collecting country-specific priority information. 
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QUESTIONNAIRE DEVELOPMENT 

IMPORTANT CONSIDERATIONS 


Using site-added 

questions 


Treating 
questionnaire 
development as an 
iterative process 


Using a consensus 
building approach to 
questionnaire 
development 


The incorporation of site-added questions in a questionnaire offers 
flexibility to sites for monitoring their own priorities. This 
translates to buy-in, participation, and support of a national data 
collection system. As local sites take ownership of their own data, 
this approach helps build capacity—both at the local level as skills 
are developed to administer and analyze results, but also at the 
national level, as more sites participate in the surveillance system 
over time. 

Remember that the goal is to collect accurate data. Relevant 
content experts need to be involved in the design of your 
questionnaire. The final questionnaire may go through five to 
seven drafts or versions before it is ready to be used in the field. 

Pilot testing the questionnaire is the last step in the questionnaire 
development process. A questionnaire development team should 
check the general flow of the questionnaire topics, the arrangement 
of questions within each topic, and the clarity and understandability 
of questions and directions. 

Establishing a surveillance system and developing the 
questionnaire(s) that will be used require many people working 
together across levels and sectors, and with staff from other 
surveys. Involve people with a wide range of experience in health- 
related issues and questionnaire design. 

Policy makers should be included in the entire survey process, 
beginning as early as the questionnaire development stage. This 
may result in a greater buy-in and use of data. 

Involve relevant survey research experts from the beginning. They 
can provide information on content, measurement, data analysis, 
and statistical support. 

Be sure that your questionnaire and surveillance system meets 
needs at all levels — from global to local. A consensus approach to 
questionnaire development builds buy-in, support, and ownership 
for those involved, and provides a good example for health 
programme and policy development as well. 
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QUESTIONNAIRE DEVELOPMENT 


IMPORTANT CONSIDERATIONS 

data are being entered. This can result in a cost savings to your 
survey. 

Determining the The length of the questionnaire itself has to be considered. As the 
length of number of questions included on a questionnaire increases, the 
questionnaire burden on the respondent increases. As respondent burden 

increases, so does the risk that a respondent might tire and either 
terminate the interview before all questions have been asked or 
provide false answers to simply get through it as quickly as 
possible. Questions placed at the end of a lengthy questionnaire are 
in jeopardy of not being completed at all or not being completed 
accurately. 

Costs associated with data collection also increase as questionnaire 
length increases. These costs are in part due to an increase in the 
amount of interviewer time required, questionnaire printing costs, 
etc. 

Breadth of topic coverage of the questionnaire also is important. 
Carefully balance the number of topics included versus the amount 
of detail required on any specific topic. 

T ranslating the The questionnaire may need to be translated into additional 
questionnaire languages depending on subpopulations in each country. Weigh 
the number of languages needed and the cost of translations. Be 
sure to consider the impact on sample size needed to collect 
subgroup data (see section on calculating the sample size). 

After the instrument has been translated, it must be translated back 
into the original language to ensure that no errors are introduced, ff 
there are regional differences in languages, be sure to test the 
translations in each of the regions. 

Independent, professional translators who are familiar with the 
issues of providing accurate translations are preferable to those with 
less experience. 
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QUESTIONNAIRE DEVELOPMENT 


IMPORTANT CONSIDERATIONS 


Designing the Check for ambiguity in the wording of questions. Try to ensure 
questionnaire that respondents cannot interpret the meaning of words differently. 
Adapt wording to the vocabulary and reading level of the 
respondents. 

Use questions that are appropriate for your country or region. 
Avoid using abbreviations and jargon. 

If necessary, add a note in the labeling of the question, specifying 
the exact meaning of a word or provide culturally relevant 
examples in the question. For example, in some areas, the word 
“wine” may be considered as synonymous with alcohol. Another 
example is the confusion between quantities such as "serving," or 
"slice, etc." 

The time frames used in questions can greatly influence the type of 
data obtained. Results and reliability will be impacted by factors 
such as the risk factor being assessed, frequency of data collection, 
and sample size. Be aware of holidays or special events that can 
occur during the data collection as they can result in unusual food 
or drink intakes, as well as different habits. Careful consideration 
should therefore be given to the time frame used, and attempts 
should be made to be as specific as possible. For those sections of 
the questionnaire that are self-administered, consideration must be 
given to issues such as font and instructions. Instructions must be 
easy to understand and stand out so that respondents pay attention 
to them. The font must be a large enough size and style for 
respondents to read easily. 

Be sure that all response categories are mutually exclusive and 
provide for all responses. 

Use skip patterns so respondents do not have to answer questions 
that do not pertain to them. If a question refers to a previous 
question, be sure to include enough wording of the previous 
question to avoid confusion. Be sure to ask questions in a logical 
order. 

If possible, keep in mind the coding scheme that will be used when 
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QUESTIONNAIRE DEVELOPMENT 



Maintaining a 
balance between 
question 
consistency and 
quality of 
information 
collected 

Decreasing 

measurement 

errors 



IMPORTANT CONSIDERATIONS 


The decision of whether or not to modify a question must be 
carefully balanced to determine to what extent question wording 
and order changes impact measurement properties. 

More information on question reliability and validity is provided 
with the Resources at the end of this section. 

What is at stake with regard to question modification is the need, on 
the one hand, to keep question wording the same over time for 
trend data analysis, and the need, on the other hand, to ensure that 
the questions are performing optimally to provide important 
information. Retain consistent question wording to the extent 
possible, and make wording changes only when a question is not 
providing clear information. 

Measurement error is the difference between an observed score and 
a true score. Measurement error can be kept to a minimum with 
careful planning and design of the questionnaire. Questions should 
be limited to one issue, and simple wording should be used as 
longer questions get confusing. 

• It is important to make sure the recall period being assessed 
can be remembered by the respondent. Words such as 
“regularly” and “occasionally” mean different things to 
different people. Use more specific words, such as "daily," 
"weekly," "monthly," "past week," or "past month." 

• Avoid wording questions so that socially desirable 
behaviour is obvious in the question or the response options. 
Consider sensitivity of questions (e.g., diet, sexual 
behaviour, drug use). 

• Questions that are acquired from previous surveillance 
surveys may introduce less measurement error than new 
questions that have not yet demonstrated reliability. 

• Consider cognitive testing of questions. 

Types of measurement error are defined and included under 
Resources at the end of this section. 
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QUESTIONNAIRE DEVELOPMENT 


Determining which 
risk factors 
to measure 


Determining how 
data will be used 



Using standard 
versus modified 
questions 


IMPORTANT CONSIDERATIONS 


The following need to be taken into account in determining which 
risk factors to measure: 

• Number and significance of associated health problems— 
morbidity, mortality. Disability Adjusted Life Years 

• Global NCD priorities 

• Policy or intervention that are available/possible 

• Ease and accuracy of collecting data 

• Risk factor prevalence levels 

Consider the purpose of asking each question with regard to how 
the data will be used. There are many ways in which data collected 
through a behavioural risk factor surveillance can be used. In 
designing your questionnaire, consider the following data uses: 

• Monitoring health behaviours in population groups 

• Planning programmes or interventions 

• Developing policies 

• Producing research findings 

• Providing public information 

• Monitoring progress of global, regional, or national health 
objectives 

The first step in constructing a questionnaire is to consider what 
kind of data is wanted or needed. It also is important to consider 
the data needs of organizations involved or impacted by the 
surveillance efforts. 

See examples of how data are used from the U.S. Behavioral Risk 
Factor Surveillance System under Resources at the end of this 
section. 

Using standard and well-developed questions from the global, 
regional, or national level module helps you avoid having to assess 
reliability and validity of locally developed questions. Further, a 
standard set of questions used across countries or locations within a 
country will enhance comparability of data. 
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QUESTIONNAIRE DEVELOPMENT 


ACTION STEPS 


information from it. 

After the questionnaire is translated, have it translated back to the 
original language, preferably by someone other than the original 
translator, to check for problems that could undermine the validity 
of the questions being asked across subgroups. It is important that 
this task be conducted by an independent interpreter to ensure 
accurate translation. 

In addition to the correct translation, cultural differences should be 
taken into account. An example from Mexico shows that “Pulque” 
is not always considered to be an alcoholic beverage. Some people 
view it as part of the regular diet. Only a quality translation will 
allow for questionnaire comparability between countries, or 
between any areas where different languages are used. 

Print the Each version of the questionnaire should be prepared as a camera- 
questionnaire ready master for printing. Sufficient time should be scheduled to 
allow for reproduction of the questionnaire copies. 

A general rule is to print 100 more questionnaires than the intended 
sample size, plus 5 copies per interviewer for use in training. This 
should provide an adequate number of questionnaires to conduct 
your survey. 

Review the Review your questionnaire between each surveillance 
questionnaire on a implementation. The goal is to balance consistency over time 
regular basis against quality of information being yielded. Carefully consider 
making changes, and make changes only when necessary. 

When feasible, cognitive testing of questions before 
implementation helps to ensure that respondents clearly understand 

a the questions and provide accurate responses. A bibliography on 
cognitive testing is provided under Resources at the end of this 
section. 
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QUESTIONNAIRE DEVELOPMENT 


ACTION STEPS 


Develop the core 
questionnaire 



Develop and select 
optional questions 

'TWtep^onS 


Pilot test the 
questionnaire 


Translate the 
questionnaire 


Questions included in the core questionnaire should reflect 
behaviours that contribute to the leading causes of death and illness 
globally, regionally, and nationally. This helps to ensure that the 
questions address the most critical noncommunicable disease 
(NCD) issues. 

Core questionnaire items also should be linked to interventions that 
are either currently available or have the possibility of being 
developed. Collecting data with no linkage to specific policy or 
programmatic efforts would have questionable utility and would 
seriously undermine the sustainability of data collection efforts. 

Questionnaires item from global and regional surveys are provided 
under Resources at the end of this section. 

Questions that are not in the core questionnaire can be used as 
optional modules. Taking respondent burden into account, a 
decision can be made as to how many NCD topics can be included 
in a questionnaire, and to what extent detailed information is 
needed for any given NCD topic. 

Leaving room on a questionnaire for sites to add questions that are 
important at the local level increases buy-in, support, and 
participation. 

Schedule adequate time to allow for a pilot test of the 
questionnaire, as well as for the survey administration process. Use 
the results of the pilot test to: 

• Check and improve question wording, directions, and 
transitions between questions and questionnaire sections. 

• Check and improve the administrative process of data 
collection, including interviewer directions and clarification 
notes in the interviewer protocol. 

The cost of translation will impact the number of languages into 
which the questionnaire can be translated. If there are numerous 
subpopulation groups in your country, balance the cost of 
translation against the cost of obtaining the sample sizes required. 

If a particular group is small in number and/or difficult to access, 
you may need to consider other approaches to collecting 
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QUESTIONNAIRE DEVELOPMENT 


Standards: 

1. Ensure that good trend data are obtained. 

2. Ensure that data collected are directly linked to 
prevention use. 
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QUESTIONNAIRE DEVELOPMENT 


Questionnaire design and development require careful thought 
so the data collected provide information that can be used by a 
wide range of audiences, including decision makers, 
researchers, politicians, and the public. 

Questionnaire development Action Steps include: 

1. Develop the core questionnaire. 

2. Develop and select optional questions. 

3. Pilot test the questionnaire. 

4. Translate the questionnaire. 

5. Print the questionnaire. 

6. Review the questionnaire on a regular basis. 


Important Considerations to take into account when 


developing a questionnaire include: 


1. Determining which risk factors to measure. 

2. Determining how data will be used. 

A -— 3. Using standard versus modified questions. 

x '4. Maintaining a balance between question consistency 
and quality of information collected. 

5. Decreasing measurement errors. 

6. Designing the questionnaire. 

7. Determining the length of questionnaire. 

8. Translating the questionnaire. 

9. Using site-added questions. 

10. Treating questionnaire development as an iterative 
process. 

11. Using a consensus-building approach to questionnaire 
development. 

12. Developing questionnaire comparability across 
countries. 




Tor 


Goals and Standards are: 

Goals: 

1. Work toward validated questionnaire sections and 
items. 

2. Minimize respondent burden. 
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